Soft++, a multi-parametric non-saturating non-linearity that improves convergence in deep neural architectures
نویسندگان
چکیده
منابع مشابه
A Non-Technical Survey on Deep Convolutional Neural Network Architectures
Artificial neural networks have recently shown great results in many disciplines and a variety of applications, including natural language understanding, speech processing, games and image data generation. One particular application in which the strong performance of artificial neural networks was demonstrated is the recognition of objects in images, where deep convolutional neural networks are...
متن کاملNeural Networks: What Non-linearity to Choose
Neural networks are now one of the most successful learning formalisms. Neurons transform inputs x1, ..., xn into an output f(w1x1 + ... + wnxn), where f is a non-linear function and wi are adjustable weights. What f to choose? Usually the logistic function is chosen, but sometimes the use of different functions improves the practical efficiency of the network. We formulate the problem of choos...
متن کاملParametric Analysis of Fiber Non-Linearity in Optical systems
AbstractWith the advent of technology Wavelength Division Multiplexing (WDM) is always an area of interest in the field of optical communication. When combined with Erbium Doped Fiber Amplifier (EDFA), it provides high data transmission rate and low attenuation. But due to fiber non-linearity such as Self Phase Modulation (SPM) and Cross Phase Modulation (XPM) the system performance has degrade...
متن کاملNon-parametric estimation of non-linearity in a cascade time-series system by multiscale approximation
The paper addresses the problem of using multiscale approximation for the identification of non-linearities in Hammerstein systems. The exciting signals are random, stationary and white, with a bounded (unknown) probability density function, and system outputs are corrupted by a zero-mean stationary random noise white or coloured. The a priori information is poor. In particular no parametric fo...
متن کاملMulti-Bias Non-linear Activation in Deep Neural Networks
As a widely used non-linear activation, Rectified Linear Unit (ReLU) separates noise and signal in a feature map by learning a threshold or bias. However, we argue that the classification of noise and signal not only depends on the magnitude of responses, but also the context of how the feature responses would be used to detect more abstract patterns in higher layers. In order to output multipl...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Neurocomputing
سال: 2020
ISSN: 0925-2312
DOI: 10.1016/j.neucom.2019.12.014